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(54) TiUe: METHOD AND APPARATUS FOR RESTORATION OF LOW RESOLUTION IMAGES 



(57) Abstract 

The definition of a low-resolution im- 
age (10) may be improved by taking advan- 
tage of characteristics of its bimodal distri- 
bution (202) so as to produce an expanded 
image (206) with improved definition. This 
can not only reconstruct image infomiation 
deteriorated by low-resolution devices, but 
also can refine original low spatial resolu- 
tion so as to significantly improve image qual- 
ity. An image is improved by iteratively solv- 
ing a nonlinear optimization problem using 
a prcfcned Bimodal-Smooth- Average scoring 
method. Among the applications for the tech- 
nique are improved optical character recogni- 
tion, restoration of binary images, and restora- 
tion of video frames. 
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TITLE: METHOD AND APPARATUS FOR RESTORATION OF LOW RESOLUTION IMAGES 

INVENTORS: Paul THOUIN and Chein-I CHANG 



This application is entitled to the benefit of the filing date of 
U.S. Provisional Application Serial No. 60/127,316, filed April 
1, 1999. 

FIELD AND BACKGROUND OF TBE INVENTION 

Field of the Invention 

The present invention relates in general to image processing, and 
more particularly to creating a higher resolution image from a 
lower resolution image. 

Background Information 

The invention described and claimed herein comprises a novel 
method and apparatus for improving the definition of a low- 
resolution image by taking advantage of characteristics of its 
bimodal distribution so as to produce an expanded image with 
improved definition. This can not only reconstruct image 
information deteriorated by low-resolution devices, but also can 
refine original low spatial resolution images so as to 
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significantly improve- image quality. «. image is improved by 
iteratively solving a nonlinear optimization problem using a 
preferred Bimodal-Smooth-Average scoring method. Smong the 
applications for the technique are improved optical character 
recognition, restoration of binary images and video frames. 

AS used herein, the terms "low" and "high" are comparative only: 
they refer to the comparison between two images and are not meant 
to imply any absolute degree of resolution. 

« described in greater detail below, image acquisition consists 
.f converting a continuous image into discrete values obtained 
from a group of sensor elements. «hile generally cheaper to 

.oduce and transmit, a low resolution imaging system generally 

.oduces a less accurate representation of the 
than would a high resolution imaging system. Co»on methods 
expanding low resolution images to high resolution images by 
interpolation typically smooth over important details rn the 
p„cess. Unear interpolation tends to smooth image data at 

transition regions, resulting in blurry images. Cubic spline 

expansion allows for sharp transitions, but tends to produce 

ringing effect at discontinuities. 

Many applications .including 0C«, video surveillance and video 
co^ression algorithms, would benefit from improved expansion 

techniques. 
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Processing oi document images has traditionally Iseen 1«!WfiKd 4»«»g--^ 
binary format due to the savinp in disk storage and computer processing. As com- 
puter performance and storage capacity have dramatically increased recently, grayscale 
scanning and processing of document images have become more prevalent. Text images 
traditionally have been obtained f.om documents. However due to the current increase 
in digital Video a significant an^ount of text is found in this type of imagery as well. 

Document images are typicaUy acquired by a scanner, whereas video frames are most 
often captured by a digital camera. The image acquisition process in both cases consists 
of converting a continuous image into discrete values obtained from a group of sensor 
elements. Each sensor element produces a value which is a function of the amount 
of Ught incident on the device. For S-bit grayscale quantization, the aUowable range 

in higher resolution imagery. 



3 



wo 00/60532 



PCT/USOO/08494 



Text image resolution expansion has become increasingly important in a number of 
areas of image processing. Optical Character Recognition (OCR) of document images 
continues to be of great importance as we attempt to become a paperless society. Restor- 
ing text from video surveiUance imagery is often crucial to law enforcement agencies. 
Digital video compression algorithms can also benefit from successful text resolution 
expansion techniques. ' Common methods of interpolation, which were not designed 
specifically for text images, typically smooth over the important details and produce 
inadequate expansion. This paper proposes a new nonlinear restoration technique for 
text images, which creates smooth foreground and background regions while preserving 
sharp edge transitions. 

There has been significant research in both the text enhancement and resolution 
expansion fields. Some text enhancement effori. focus on fixing broken or toudung 
characters [l] 12). Other methods [3] [4] use degraded sanxples to model the image and 
then solve for the restored image using using a recursive technique. To remove the 
effects of noise. [5] uses a morphological filter and 16) use, pixel patterns to improve 
OCR results. A variety of methods have been proposed in order to improve contrast 
Within text images indudingquadratic filters l7l.softmorphologicalfiltersl8].non4in^^ 

capping 19]. and using a multi-resolution pyramid approad. and fuzzy edge detectors 
(10]. Numerous resolution expansion methods have been published in the literature as 
well im-ln). Linear interpolation tends to smooth the image data at transition regions 
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and results in a high-r^lution image that appears blurry. Cubic spline expansion allows 
for sharp transitions, but tends to produce a ringing effect at these discontinuities. 

A number of research efforts investigated combining text enhancement with resolu- 
tion expansion in order to improve low-resolution text images. Shannon interpolation 
is performed with text separation from the image background in [18] to improve the 
OCR accuracy of digital video. Deblurring is combined with linear interpolation in (191 
to enhance document images obtained from digital cameras. Both of these methods es- 
sentially perform the text enhancement and resolution expansion as two separate steps. 
The proposed method computes resolution expansion using an algorithm specifically 
designed to enhance text images to perform both of these steps simultaneously 

The goal of resolution expansion is to create an expanded image with improved defi- 
nition from observed low-resolution imagery. Acquisition of this low-resolution imagery 
can be modeled by averaging a block of pbcels within a high-resolution image. Resolution 
expansion is an ill-posed inverse problem. For a given low-resolution image, a virtually 
infinite set of expanded images can be generated by the observed data. 
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SUMMARY OF THE INVENTION 

The foregoing problems are overcome, and other advantages are 
provided by a. method and apparatus for enhancing an image by 
iteratively solving a nonlinear optimization problem using a 
preferred Bimodal-Smooth-Average (BSA) scoring method, in 
accordance with the invention, which minimizes a BSA score, 
defined as the weighted sum of separate bimodal, smoothness and 
average measures. This can not only reconstruct image 
information deteriorated by low-resolution devices, but also can 
refine original low spatial resolution so as to significantly 
improve image quality. 

It is an Object the invention to restore a high-resolution 
image given a low-resolution image. 

Xt IS an object of the invention to provide a method and 
apparatus for improving the definition of a low-resolution image 
so as to produce an image with improved definition. 

It is a further object of the invention to provide a method and 
apparatus for reconstructing Image Information deteriorated by 
low-resolution devices. 

It IS a further object of the invention to provide a method and 
apparatus for refining original low spatial resolution so as to 
significantly improve image quality. 



6 



wo 00/60532 PCT/USOO/08494 

It is a further object of the invention to auto,.ati=ally enhance 
text within document or video images so as to improve the 
accuracy of subsequent optical character recognition. 

It is a further object of the invention to provide a method and 
apparatus for improving restoration of video images. 

. principal feature of the invention is the iterative solution of 
3 nonlinear optimization problem using a preferred Brmodal- 
Smooth-average scoring method. 

^ong the advantages of the invention are automated enhancement 

of images . 

These and other objects, features and advantages which will be 
apparent from the discussion which follows are achieved, in 
aLdance with the invention, by providing a novel met ho and 
apparatus for improving image ^ality by iteratively solving 
Tnlinear optimization problem, using a Bi^dal-Smooth-.verage 
scoring method. 

.he various features of novelty which characterize the inventi^ 
are pointed out with particularity in the claims annexed to and 

rm ng a part of this disclosure, .or a better understanding of 
te invention, its advantages and objects, reference is made to 
the accompanying drawings and descriptive matter in whrch 
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f erred embodiment of the invention is illustrated. 



BRIEF DESCRIPTION OF THE DRMJINGS 

The foregoing and still other objects of this invention will 
become apparent, along with various advantages and features of 
novelty residing in the present embodiments, from study of the 
following drawings, in which: 

Figure 1 is a schematic of a system suitable for carrying out the 
invention. 

Figure 2 is a block diagram of the text resolution expansion 
system of the invention. 



Figure 



3 is a flow chart of the BSA process of the invention. 



Figure 4 is an Illustration of the results of a BSA restoration 
experiment, showing the progressive improvement of resolution 



with iterations. 



Figures 5 illustrates the performance of the BSA process as 
compared to prior art techniques of linear interpolation and 
cubic spline in grayscale document restoration and grayscale 
video frame restoration, respectively. 
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Figures 6 and 7 illustrate the performance of optical character 
recognition following cubic spline (Figure 6) versus BSA (Figure 
7) image enhancement. 

Figure 8 compares OCR accuracy of spline vs. BSA for gray-level 
images and binary document images. 

i^^Y.*■^^ results of binary document 
Figure 9 illustrates experimental results or 

restoration. 

ng«re= 10 and U Illustrate experimental result, of restoration 
of video frames. 
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DESCRIPTION OF THE PREFERRED EMBODIMENT 

The invention i. a novel method and apparatus which improves the 
definition of a low-resolution inage by taking advantage o£ 
characteristics of its bi-odal distribution so as to produce an 
expanded i.age with in^roved definition. This can not only 
reconstruct i.age information deteriorated by low-resoXution 
devices, but also can refine original low spatial resolution 
as to significantly improve image quality. 

Referring to Figure 1. the invention may be illustrated in the 
context Of an image acquisition system (1, providing .mage d t 

general purpose computer (100) 
110) which serves as an input to a general P 

programmed to carry out the following processes. 

image acquisition system ,1, comprises a plurality of image 

Tn ors 3, which generate image data .10, concerning the image 

. data (10, serves as input to computer (100, v.a input 
This image data (lUJ setv 

means .101,. Computer ,100, includes processing means 102,, 
progranm>ed to execute the steps set forth herein so as to 
generate a high resolution Image, which may be displayed or 
otherwise used via output means (103, . 

0 s text resolution expansion system 
Referring to Figure 2, a texi; 
ecelves low-resolutlon Image ,10,. preprocesses the Image in 
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estimation module (202); the bimodal estimation module (202) 
processes the data and outputs the result to resolution expansion 
module (203) for pixel replication and output to BSA restoration 
module (204) where the image is partitioned into overlapping 
bloocs prior to BSA restoration; the BSA restoration module then 
outputs the result to a post-processing module (205) which 
provides an expanded resolution image (206) for display or other 



use . 



The operation of the BSA restoration module ,203) is illustrated 
in greater detail in Figure 3. An initial image (301, rs 
provided and an initial BSA score is computed (302); an image 
update is then computed (303, so as to produce an updated imago 
,304, A new BSA score is co:,*uted for the updated image (305, 
and compared to the prior BSA score (306); if the new score does 
not represent an improvement over the prior score by a 
predetermined amount, the restoration process is complete; 
otherwise, another image update is computed (303, and the process 
continues iteratively until the improvement is below the 
predetermined amount. 
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Th. image aculsltion proofs cosi«s of cooverUw • "-'^uou, image into discrete 
obtained from a group of sensor dements. E«i sensor element produces a vine 
„Uch is a function of the amourrt of Ught incident on the device. F>.r 8-bit . grayscale 
,„anti.ation, tbe allo^bl. range of value, f.. each sensor are integers from 0 (biacK) 
„ 255 (*t.). The sensors are typically i» a non-overlapplng grid of scuare 




. waller elements result in higher resolution imagery, 
cements, »»aUerele ^ „f ^.nsor. U adcuat. to represent 

U,h.resol«tion imaging system^ the numo 
' „C pbcels within the image are eith«wh.te or^ 



U,e desired t«ct im.«e. The majority 

.U* »ith a small number of grsy pbtels occurring at the edges 
low-resolution imaging system* 



the number of sensors has been reduced^ 

. This low-resolution acquisition 





results in signiacant blockiness and is 



insufficient to accurately represent this image. 



element effectively averages the image 



withux its section of the grid, resulting 



'7,^, .^elsLow-resolution imaging can therefore be tho^^^ 
in an increased amount of gray pixels, bow 

of as block-averaging high-resolution images. 



The problem I 



>is to restore the high-resolution image JfW 
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given 



only the low-resolution image U,.. where r and c are thrdtoBef dW^ anlT 
columns in the low-resolution image and , is the resolution expansion factor. The image 
acquisition process of obtaining Llr^ from H V^c is given by 



1 (gr-K-l) 
9 #=9T t=9-c 



(1) 



The value of is the average of the high-resolution pUels within the , x , neigh- 
borhood. EC. (1) represents a typical image restoration problem where we are re<,u.red 
restore the HV. based on the observed L/. via the.relationship described by this 
equation. Since there are a great number of high-resolution images whid> may satis^ 
..e constraint of the observed tow-resolution image given by Be, (D. in^age restoration 
is generally an iU-posed inverse problem. 



to 



i'he system presented in^ 



block diagranJHMiin Fig. ^Jjprcomprises five modules, the Preprocessing 
Module, the Bimodal Estimation Module, the Resolution Expansion Module, the BSA 
Restoration Module, and the Postprocessing Module. A low-resolution image U initially 
input to the Preprocessing Module where binary and color images are converted to 
grayscale images. The image histogram is computed by the Bimodal Estimation Mod- 
ule to estimate the means of the black and white pixel distributions. The initial image, 
expansion is performed by the Resolution Expansion Module which uses pixel replica- 
tion to create a high-resolution image from the low-resolution original. The image is 
divided into overlapping blocks of pbcels which are restored independently by the BSA 
algorithm in the BSA Restoration Module. When a binary expanded image is desired, 
the Postprocessing Module performs a grayscale-to-binary conversion. The next section 
details the image expansion system for grayscale images, followed by a section on binary 
image expansion. 
' 3 Grayscale Image Resolution Expansion 
The image resolution expansion process for grayscale images, which includes both 
video frames and grayscale document images, is described in this section. It should 
be noted that our system depicted in Fig. ^processes grayscale and color images m a 
. very^^lar fa5hionr.^:Rw^ly^differ^ce> iSim MSdmr^KSre-^^br 
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.e .m^y -venea to ^y^e ^ techni^u^s. The. 

^ — ' " ^'-^ ""^ °' " 
„ ^p,.e — ea oca .e.,., ^ o, co,o. — . not . o.nce». 

,«..Uo„Moa.U. THea.*uUo„ orate., una. .U„.a.«..a.»se 

1 ^ — . . - - ^ - : 

„ te.t ^ oe t.. «.te ..t..t.o. a.a t.e . t^e 

a.....o„.«e«t..atea^.tHec...tea^-^-.a,e.,.^^^^ 

.... ^ t.. --0. «>.«e t. o. ea. 

:.:.co«.o.a>..— 

to .0. ..a. ..a^aea .a. BS. ^.at.a Moa„t.. 

1 .a. .to o,.,app>. UOC ana .esto.. t.e Uoe. . - 

Po^tP^oee^ Moa.e co„^ «>e -a.e to a ..a., ...e 

„..^«pa.aea..a..aes....e::n,.t.no™,..BSXKe^.t.o.Moa.e 

will be described in detail. 

. ^ aa^an. o, t.e BSA B..o«t.on ^ >■> ^-^-^ - 

• ■ is the score at iteration i, M.a «. i« the change 

^ts the image at iteration ., BSA « the scor 

. . e at it^ation i. The restoration procesa can be ,ununari»a as ioliows. The 
to the image at Iteration ». x *v,:„o„ 
...scoreBS.isco.nP„tea.on.theori.na.in.a... M each .t^atron the 
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BSA score until a convergence is reached. At this stage, a restored image is produced. 

The scoring function used by the BSA Restoration process is designed to measure how 
well a group of pixels within an image represent the desired properties of a text image. 
This function, referred to as the BSA scoring function, is expressed as the weighted sum 
of a Bimodal score B, a Smoothness score 5. and an Average score A. More precisely, 
the BSA score is defined by 



BSAix) = XiB(x) + X2S{x) + XsAix) 



(2) 



where A. A,, and A3 are Lagrange multipliers and x is a block of pixels to be restored. 
Throughout this paper, a block x is defined both as a group of 4 x 4 low-resolution 
pixels or as the 4g x 4g high-resolution pixels that are derived from them. The 4 x 4 
size was specifically chosen because it contains enough pixels to adequately measure text 
characteristics but is not too large to be computationally burdensome. The goal of the 
restoration process is to iteratively solve for the block of pixels x that minimizes the 
BSA(x) score given by Eq. (2). Each of the three scores will be bfgiy discussed in the 
following subsections followed by a description of an iterative minimization procedure. 
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.4. The Bimodal Score 

The typical distribution of a text image contains two peaks, a large one at fMwhite-. 
which normally represent the page's background, and a secondary peak at AiwocJt rep- 
resenting the foreground text Jltlf/j^gt/ggl/^ FVom the histogram of the given 
low- resolution text image, estimates of the means for the black and white distributions 
are calculated. These means are used to compute the bimodal score B(i), which mea- 
sures how far an image block x is from bimodal. The bimodal score used in this paper 
is defined by 

where r and c are the row and column indices within the block being evaluated. 

The bimodal function for nuack = 50 and fi^sae = 255 is plotted in Fig. 4. When a 
pixel value within x is close to either fiuack or /i,„Mte, its contribution to B(x) is minimal. 
The bimodal minimum score of = 0 means that the image is perfectly bimodal, 
the value of every pixel is equal to either /z„wte or yu«„e*. Solving for the block of pixels 
I that minimizes B(i) produces a strongly bimodal image, which is one of the desired 
properties of this proposed text restoration technique. 

To minimize both the first and second derivatives of the bimodal score will be 
computed. The score given by Eq. (3) is differentiable, the partial derivative of B{x) 
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with respect to pixel Xr,c is 

(4) 



and the second partial derivative for the bimodal score is 

partial derivativesofthehirnodalscoreB(.)arei„dependent of neighborin^pixeis.^ 
estimated means of the bimodal distribution. ..^ and are Unown a priori and 

their appropriate constants can be pre-computed. 
B. The Smoothness Score 

W.. ..e exception of edges. t«ct ^ tend to be very »oo«. .n bot. the .ore- 
^„nde.dbad,groundregion.,Mcbresnl.»lnne^bo.^.bsin,n3.va.«es.A=n,ootb- 

ne. wMcb . computed <b. e«b b,oc. of p.«U. . ^troduced to n.e,su,e tMs 

,e.t„«. For tb. proposed -sorUbm, a s,n,ple statistic »sb. on,, .be fo. ne^ 
neighbors of e«b pixe. is used. Otber niore sophisticated sn.ootbne,s mes^res conid 

..™ used bv this technique is given by 
be implemented bs vreU. The smoothness «»re S(x) used by 

„,ere . jc .« the ro. and column indices »ithin the bloci. being evaiuated. The 
^,mum viue of S W = 0 occurs when 11 pixel, have identical values. 

The tat and second partial derivatives are calculated h. a straightforward manner. 



17 



wo 00/60532 



PCTAJSOO/08494 



The first partial derivative of this smoothing score ..ith respect to pixel a:., is 

dXr^c 

The second partial derivatives are nonzero only when 

3nd are e,n.l « «ro .or all other con>hinaUons. The arst partial derivative of the 
^oothness score i» a ft.ncti.n or the .o. nei,hhorin, pixels. The second partial 
derivatives are non-«ro oni, with respect to their corresponding neighbors. 

Q The Average Constraint Score 

„ reasonable to re,„ire that the average of a group of high-resolution pixels is 
eiose to the original va^ue of the low-res^ution pixel fron, which they were derived. 

each bloc, of low-resolution pixels, an average score M.) is used to ...sure how 
^, .he restored high-resolution pixels meet the a^rage constraint imposed by the. 
corresponding. ow-resolution pixels. Pig. 5 shows four low-resoluUon pixels whose values 
are «.a The , x , group of high-resolution pixels that a. being restored 

Kv ItW 1 < (r. c) < «} . The average score for this 2x2 
from pixel Mi are represented by {i^^ , i S ^t". _ 

block is expressed by 

. 1 ;o value of each low-resolution 
where i is the index for the low-resoluUon pnceis, « » the value each 

pixel and x™. are the restored high-resolution pixels corresponding to pixel The 

high-r^olution inrage fornred by using pixel replication alwsys has an average 
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score of zero because it satisfies the constraint. 

The first partial derivative for the group of high-resolution pixels corresponding to 
pixel is equal to 



The second partial derivatives are simply, 

d'Mx) _1 V(r,c.n,ci)€a:« 

BO.K U,e «r« ..a «co.a ^ derivatives C the av«a.e are _ only 

with ,e.pe« to pixe. witi.io the , X , .roup o, high-resolution pixels corresponding to 
the low-resoluUon pixel ftom which they «ere exp«ided. 

IV. SOLVING FOR THE RESTORED IMAGE 

The goal of the restoration aigorithm is to solve for the image hlo* that minimises 
U,e scoring function SSM.) introduced in E,. 2. Throughout this paper, a block X is 
ae^ned both as a group o, 4 X 4 low-resolution pixel, and a. the 4, x 4, high-resolution 
puels that are derived from them. The 4 x 4 si. «as speciBcally ^csen hecause it 
eontains enough pixels to ade,uately measure text charac^ristics but is not too large 
,„ he computationauy burdensome. Tl.e goal of resolution enhanc«n.nt is to create a 
.estor^ imsge with improved resolution. This is iUustrated in Fig. 6 .here a 4 x 4 
HOC of low-resolution pixels Is expanded 1. a factor of , = 4 to create a 16 x 16 bio* 
of high-resolution pixels. 
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Pixe. replic«io„. where every vlue withiB a , X , neighborhood is identical to the 
corresponding low-resolution pixel, is used for the iniU^ expansion. Each 4,. 4, blodc of 
high-resolution pixels is restored Independently using iterative optimisation techniaue. 
described in this section «, solve for the block which minimizes the BSA s»re. At each 
iteration, the first and second partial derivative of the BSA scoring function are used 
to determine the image update. To avoid bloc, boundary discontinuities only the center 
3,X3, pixels, which arehl^tedinFlg. 6. are updated. The entire Image is therefore 
aivided into bioc^ that overlap by one quarter, or , X 4, pixels, and can be restored 
independently, ^s iterative minimisation of the BSA score continues untU convergence 
is reached resulting in the restored image. 

InitiaUy, e«=h 4, X 4, blod. of pixels . is converted to a (4,)Mong vector * using 
raster scanning, 

il,(,_l) + cl = l(r,c) for 
A small distance away from * the BSA function can be represented by its second order 
Taylor series approximation [21], 

BSA{S + S) ^ BSAi5f) 4- [VBSA{£)]U ^S^HS (13) 

and the change in BSA is given by 

ABSA = [VBSA{£)]S+ \^H5 (^^^ 
..ere S is the small change to the image vector VB5.(.) is the gradient, and H is 
..e Hessian matrix. The (4,)' x (4,)^ Hessian given below by E.. (15) is the symmetric 
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matrix of mixed partial 
affects the BSA function. 



ial second derivatives, which shows how a change in two variables 



a^BSA 
dxidx\ 



Bxi 



B^BSA 
a^BSA 



a^BSA , a^BSA 



a^BSA 



Since the Hessian matrix is symmetric, 

ff'BSA _ d^BSA 



(15) 



(16) 



dxidzj dxjdxi 

^ ot the .... needs t6 be ^ -^inO. the func«o. BSM^. the 

U. U.e Hee.. .«*c a« ..epenae.. ao *e Hess^a. . 

^ a «3^- ^ e^senveco, the Hes.. ».a.. .s 

of ..e e.^«»«ix With i« transp^e U e^a. « *e «en«.y -nauix = /■ When 

... „e.>.n .atHx .s pre-n.^ h, «.e ^.o^ 

. ■ PTHP is diaeonal. Because the Hessian is 
hj, the eigenmatrix, the resulting matrix ITHB « diagona 

^ „a symmetric, it is .1^ ^ ^ — ' "'^^ ' *' 

H^ian matrix .r an S . S hioC . pixels. Because . the neighhorhood dependence 
. the scorh. .nctions, their contributions to the Hessian are near the main diagona.. 

, J. 15,^,1 Hp«ian E^HE, which is shown at 
The similarity transform results in the d»agonahzed.Hess:an. E 

right. 

T^eT^yiorseriesappr^imationtothechangeinthescoring — AB5^ca«now 
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be expressed in terms of the (4g)» x (4g)^ Hessian matrix H, its (4,)^ x (4g)' eigenmatrix 
E, the 1 X iAg? gradient of the scoring function VBSA{£) , and the (4,)' x 1 small change 
in the image vector 5, 

ABSA = {[VBSA{x)mE^^ + \{rE){E^HE){E^5) (17) 

With the following substitutions, 

. VBSA'{£) = [VBSA{S)]E ^^^^ 

H' = E'^HE 



(20) 



Eq. (17) can be simplified to 



(22) 



The fimctionsl mimmum is Khiewd by stepping to the direction 

i" = H'-'VSSA'(f) 

in the tr^o™.ed dom«n, which is simply S = in the pixel domain. For e«h 
iteration, the image update i> is detenntoed. The iterations continne „n«l convergence 
is reached, resulting in a desired restored image. H" 

An ex«npleof this iterative imagerestoraUonprocessisshowntoFig^Theoriginal 

4 X 4 block of pixels is expanded by a factor ot , = <^^^ Pixel repUcatio. to produce 
a le X 16 highjgolution to«ge shown in Pig* As the iterative restor«ion process 
proceeds inTit Jftf. t-e image becomrmore bimodal and smooth resulting in a 

A 
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greatly improved image. The majority of gray pixels that occur between characters are 
replaced with either black or white values, resulting in a strongly bimodal distribution. 
The resulting image is also smooth in both the foreground and background regions while 
maintaing the constraint that the average of each 4 x 4 block of high-resolution pbcels is 
close to the original value of each corresponding low-resolution pixel. The minimization 
procedure is completed in 30 iterations for this 



image-i 



more 



As the iterations proceed, the image becomes 
bimodal and smooth and these two scores are reduced.! 




The average score increases as 



the restoration proceeds, but this score is still 



significantly smaller than the bimodal and smoothness scores. Minimization of the 
BSA score produces a restored image that is the optimal combination of these bimodal. 
smoothness, and average measures. 

V. EXPERIMENTAL RESULTS 

The proposed BSA restoration algorithm was compared to several common expansion 
methods, including pixel replication. Unear interpolation, and cubic spline expansion. In 
linear interpolation, a linear fit is calculated between all pixels within each column, and 
repeated for all pixels within each row. These images naturally tend to be smooth. 



then 



without sharp discontinuities 



ies, producing blurry results. Cubic spline expansion [12) 
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.ppr<»dma«s the given aiscrete low.r«olutlon pixels » a smooth continuous curve ob- 
tained from the v,eighte<i sum of cubic spline bssis functions and resamples the curve to 
obtain the high resolution Image. This method aUows for sharp edges but often over- 
shoots at these discontinuities, producmg a ringing effect. The BSA te:ct restoration 
technique c«ates smooth foreground and b«*ground regions and permits sharp edges 
« transition regions, while mah-taining the ,„«.re,o.ut,on average constraint. Images 
restored with this t»hni,ue are shown to be both ,u^itative.y and quantitatively sup. 
Hor to other conmon resolution expansion meUrods. Two experUnents were conduced 
to numerically compare the restoraUon methods. The first experiment involved creat- 
ing low-resolution images from high-resolution origin.^, expanding the low-resolution 
imagery, snd th«. measuring the dist«.c to the originals. The second experiment in- 
™„ed scanning low-resolution document images, expanding the images with the various 
techniques, and using OCR accuracy to measure the success of the restoration. 

^ quaUtatively illustrate the difference. b«we«. U.es. resolution expansion trf.- 
niques, Fig^hows resulting images obtained from linear interpolation, cubic spline 
exp»«lon, and the proposed BSA-based restoration technique. The w^d "appUcatlons" 
from an image scanned at 100 dpi using S-bit grayscale quantisatton is shown in Fig. 
^) wh.« Significant blocjdness is apparent. Linear interpolation.by a f^or of four 
„ss used to create the image h> Fig.^b) which is very blurry and lades good contrast. 
Fig.i(c) depicts the resulting image from cubic spline expansion which has better con- 
t^t but is stm not sharp at ti>e edges. The ima^ obtained using BSA restoration in 
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y . «4 is superior to the Images obcamed 

These resulto dearly a«« , . :„ pte HW i* s*«>"«ly 



feom the University 



25 



wo 00/60532 



PCT/USOO/08494 



Fig. ^) has excellent contrast and sharp edges and is superior to the images obtained 
using other interpolation methods for this example. 

The first experiment to quantitatively measure image restoration success involved 
creating low-resolution images by block-averaging images as described by Eq. (1).- Re- 
stored images are then compared with the original to determine the success of restoration 
numerically. The mean squared error (MSB) was used to compare the various methods 
of image resolution expansion. The definition of mean squared error, 



IS 

j^SE = E iZifyriginalrfi - restoredr,c) 



(23) 

where R and C are the number of rows and columns in the images. 



These results clearly demonstrate several advantages of this technique that was de- 
.gned specifically for text images. The BSA^restored image in Fig. U(e) is strongly 
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used. These binary documents were initially printed by a laser printer witrSOCTaots p^r 
inch (dpi) resolution. Each page was then scanned using 8-bit grayscale quantization at 
75 dpi to create low-resolution original images. These 75 dpi resolution pages were then 
expanded using various resolution expansion methods by afactor of four to create 300 dpi 
images which were processed by Caere's OmniPage Pro 7.0 commercial OCR package, 
the world's best-selling desktop OCR software. For all cases within these experiments, 
both spline interpolation and BSA restoration produced superior results to linear in- 
terpolation, therefore the linear interpolation results will not be discussed further. The 
resulting text files were compared to the ground truth provided by the University of 
Washington CD-ROM using the OCR Accuracy R.po'rt Version 5.3 software developed 
at UNLV-ISRI. There were a total of 339,575 characters in these 122 images. Cubic 
spline interpolation resulted in 36.959 character errors and the BS'A-based method had 
30 668 character errors for an overall improvement of 17.0% for this set of images. 

OCR character accuracy for all 122^document images is plotted in Fig. ^for cubic 
spline interpolation ^ BSA expansion. The results are sorted based on the spline OCR 
results which are shown as a solid line. For each image, the BSA result is plotted as an 
"X" if the OCR accuracy is worse than spline a«d plotted as an "O" where the accuracy 
has been improved. The BSA restoration resulted in improved OCR accuracy for 72% 
of the images in this test set. Even in the cases where cubic spline resolution expansion 
improved OCR more than the BSA algorithm, the images produced by the BSA were 
typically more visually appealing. if/g/g/gf^Kt^K^^^^^^^^^^^"^ 
^^^^^^^^^gggf^glf^gg^^ftKKf^ For the images in experiment, 
the expansions produced by the BSA algorithm, produce superior OCR accuracy results 
compared to other existing methods. 
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Binary Image Resolution Expansion 
Docun^ent images have „adiUo„al., baen canned usi„ binary thr»ho.di„g dua « 
.He inherent binary nature of text in-ages. Binary in,age proce«U>g tberetore sti.. retains 
,.at s,gniaeance in tbe document r^earoh — ity. Tbe BSA restoraUon n>eti,od 

..pabieotrestoringbinaryin-ages. in order tor onrsystemtorestore binary text ..ages, 

^ origina. binary i»a.e.arst convoked »itb a spaUa. n..a. in tb. PreP.oc^>. 
M0dn,= Shown in Fif l^^-" a grayscaie in.age. THe Bi^oda. Bs«n.ation, Kes. 
«on Expansion, an^BSA lUstoration Modu.es used for binary images are identica, 

«t«m Once the high-resolution grayscale 
,„ those used for grayscale images m our system. Once the g 

.mage has been created, a gioba. threshold r_ is used in the Postprocessing Module 
„ convert the grayscaie image bac. to . b.ar, .n«ge. This threshoid . computed to 
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,e Mfwa, be««=n the whit. ^ bimod^ ^ 



TWs section will deUil the eo-version from a bin«y 
spati^ convoiutio. rnesV iB the PxeProce»l«« Module 



Twnary — 2 

image to a grayscale image using a 



4.1 Binary to Grayscale Conversion 

orieinal low-resolution 
The««ts.epot.hebin«yre.to«ti„nproce«i.tocon,e«theon^ 

binary image into a lo«-resolut.on grayscale >m g 
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1 convolution of two images /x,y and 5x.y, 



.o. 0 (Mac.) .o 25S (wMte). Co— n is perfonned to ceate .a. values .o. a 
group of neighboring binary pixels. The discrete . 
denoted by U,y * is defined by 

R-l C-1 (10) 
fr,cSx-r,v-e 



• - . it must be synunetric about its center, 
xnask. Because convolution involves flxppxng 5... »t must sy 

Throughout this paper the 3 x 3 mask fl,. defined by 



wc '^s 

Ws 

ty(7 lt;5 ""^C 



. Of «s a weighted averaging over a neighborhood of 
Spatial convolution can be thought of as a weight 

, >.Med on the distance from the center 

p^*. A constraint U enforced on th«e «««h.s, based 



corner 



30 



wo 00/60532 



PCTAJSOO/08494 



of the mask, 

v^M >ws>wc>^ (^^^ 

so pixels closer to the center are more heavily weighted. The value of the grayscale 
pixels x^^'^y are easily computed from the original binary pixel Xr.c along with its eight 
neighbors 



4.2 Consln-inrf Binan/ to CW»ie Con.crsio» 
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,» r>,. SW . " ^* 

Aj. (14) 

... or.. - ^ - ^ - — ' 

^ on OH,.. ... ^ - - roUo.ns 

- V 1^1 (15) 
if 1 = 0 then i»"''(r.c) = l-2 J 

if x = 255 then x»'«v(r,c) = 128+ l| J 
...e l.J is the integer floor — g CuncUon defined W the latest integer .hac . 

less than or equal to x. 
To quantitatively compare our system with other expansion methods, resulting im- 
ages were rompared with cubic spUne expansion to measure OCR performance. Shown 
in Fig. the resulting 300 dpi image obtained using cubic spline expansion on 

a paragraph from one of the Uni^ity of Washington document images. The text file 
created by OCR is shown in Fig. ^Cb) where mistakes have been highlighted. For this 
example, the OCR results for the cubic spline image had 7 areas where mistakes were 
made. The same image paragraph was processed by our system resulting in the image 
in Fig. |(7which has better contrast than its corresponding cubic spline image in Fig. 
^(a). O^R results are improved as well, only 4 mistake areas are highlighted in Fig. 

^(b). 
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PAGE 33 INTENTIONALLY OMITTED 
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Experiment — University of Washington Binary Documents 

To n,ea.u,e the success oi the system to V«t=H« bto»y document images, a second 
experiment was conducted using a total o. 48 document pages from the Unive^lty of 
Washington ia.ab.se. These images were scanned at 100 dpi using binary ouantisation 
and then expanded by a factor of three using the binary BSA restoration technique 
aescribed in Section 4 to create 300 dpi images. The binary enhancemen^process for 
.he sample word ..repres«,tative... from this experiment is shown to Kig. » to visuaiiy 
...strate the improvements. The originai binary image to Pig. «■) - convoived 
w,th a spati- maa,. of weights = : and »c = 0.T0T to produce a grayscale image .„ 
Pig J(b). This iow-resoiution grayscale image was enhanced by the system to produce 

1, im.« to Fig TkO »hich was converted to a binary .mage 
a high-resolution grayscale image in til. WK'l 

■ o- The resulting high-resolution image was 

to produce a restored image m Fig. The resmt g 

noticably less blocky than the original. 

TO quantitatively measure the success of the system in enhancing binary document 
images, OCR accuracy was compared before and after resolution expansion. The ex- 
panded images pr oduced by our system were compared with .00 dpi images created^ 

pixel replication. PlM^^^^^^^^^^^^ 

There were a total of over 140,000 characters in this dataset. The overall 
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ch««ter accux^ for the origima unices was 82.0% and the ov«,ll cb^arter «xor«y 
for the rertored images was 89.1% which was a 39.5% reduction in e«o». * significant 
improvement over pixel lepUcation. T'lft ^ ) 
5^ System PerfomumeA 

The previous experimental xesolts dcmonsttate the ad^tages of our text restoration 
system compared to standard methods of interpolation such as linear interpolation and 
cubic spUne expansion. n.e system was tested using imag« from the widely available 
Unive»ity of Washington document database and evaluated using the commercial Caere 
OmniPage PJo OCR aoftware package to measure character accuracy. For the document 
aad video images in these experimeiits. the images produced our restoration s>^ 
resulted in a higher OCR accuracy compared to other standard methods. 

As stated earUer MHM. the goal of this resolution exp««don system is to 
improve the OCR accuracy. Real-time image proces«ng is not a reqain«nent. Our 
system approximately 10 minutes to expand a single full-page 100 dpi document 

i^e by a factor of 3 on a 250 MH« woAstation- Cubic spline interpolation is much 
faster than our system and take, about 10 seconds for a sin^ document pag*. Video 
fr«ne, are typi«Jly smaller than lo^resoluiion document images «xd are therefore 
processed more qmddy by the system. A 320 x 240 video frame takes approximately 
2 minutes for our system to expand Vjr a factor of three. Cubic spline exp«udon of a 
video frame only takes about 2 seconds. H the speed concern becomes issue in the 
fhture. our approad. has a potential for massive paraUeUzation because the images c«. 



35 



wo 00/60532 



PCT/USOO/08494 



be divided into 



blocks of pixels which can be restored iiidependentty. 




'rL, system shown to be capable ot enhancing grayscale 
documents and video images a. weU as binary aocum«« imag-. The snccs. or th. 
^«m was demonstrated ^ experiments using images from a standaH document im- 
,e database and a comme^ia. OCR P»*age. fUstoration of grayscale images was 
^„„ed b, opt.mi.ng bimodai, smoothness, and average <BSA) scores that measure 
desired properties o, text images. The« scores were combined to form a single scoHng 
.action Which produced images that were strongly bimodai and smooth, whiie satisfy- 
the average co^traint score. When the original image was binary, it w. initially 
convened to a grayscale im^ using a spatial convolution mssi. and then processed as 

^ayscale images. These resultant images restored using our system were shown to be su- 
peHortoimagesexpandedusingexistinglinearinterpolationandcubicspUneexpansion 

techniques. 

Op.ic.1 charac^ -gniU- accuracy, was al. used to auantitativeiy measure the 
..cess of various resolution expansion methods. Both binary snd greysCe text imag. 
.astoredwithour system wereshownexperimentallytoimproveOCaaocuracycompared 

.imearinterpolationandcubicsplineexpansion. .espite that our system wasdesigned 
Chance text images.it can .isobeusedtoaccurately».P^achec.. tables. Chart. 

.aphs. iine drawings, and other types of documents as well due to their similarities. 
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US provisional Application Serial No. 60/127,316, filed April 1, 
1999 (including its attachments) and the following attached 
documents are incorporated herein by reference as if fully set 
forth herein: 

1 "A Method for Restoration of Low-Resolution Bimodal Images" 
(authored by CHEIN-I CHANG and PADL THOOIN, unpublished, but 
scheduled for presentation to 1999 Symposium on Documentation 
I^,e understanding Technology at Annapolis, MD April 14-16, 
1999) 

, -constrained nonlinear restoration of JPEG compressed. low- 
resolution text from grayscale images using a Gibbs-Mar.ov random 
field prior", authored by CHEIN-I CHASG and PAUL THOUIN, and 
published in IS.T/SPIE 10th Annual Symposium, Electronic Imaging 
.98: science . Technology (presented at San Jose, California 
January 24-30, 19981; and 

3 "A Glbbs-«arlcov Random Field Approach to Restoration of JPEG- 
c„pressed Text Images" (unpublished, authored by CHEIN-I CHANC 
and PAUL THOOIN) . 

While a specific embodiment of the invention has been shown and 
described in detail to illustrate the application of the 
principles of the invention, it will be understood that the 
invention may be embodied otherwise without departing from such 
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principles and that various modifications, alternate 
constructions, and equivalents will occur to those skilled in the 
art given the benefit of this disclosure. Thus, the invention xs 
not limited to the specific embodiment described herein, but is 
defined by the appended claims. 
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CU^XMS 

We claim: 

1, A process for restoration of a low resolution image 

comprising the steps of: 

subdividing said low-resolution image into groups of pixels ; 

generating initial conditions as an initial expansion of a 
restored (high-resolution) image; 

iteratively revising each group of the restored (high- 
resolution) image so as to minimize the BSA score for said 
group of pixels . 

2. The process of claim 1 wherein the step of generating initial 
conditions comprises the step of either replicating or linearly • 
interpolating or cubic-spline interpolating each group of pixels.. 

3. The process of claim 1 wherein said low- resolution image is a 
binary image. 

4. The process of claim 1 wherein said low- resolution image is a 
gray-scale image. 

5. The process of claim 1 wherein said low-resolution image is a 
color image. 
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6. The process of claim 1 wherein said low-resolution image is a 
video image. 

7. The process of claim 1 wherein an image is provided, an 
initial BSA score associated with said input is computed, an 
updated image output is produced from said image and its 
associated BSA score, an updated BSA score is computed associated 
with the updated image output, the prior computed BSA score is 
compared with the updated BSA score and updated image outputs and 
calculate updated BSA scores are produced iteratively until an 
updated BSA score does not differ from its prior computed BSA 
score by more than a predetermined amount. 

8 A machine for restoration of a low resolution image 
comprising image acquisition means for providing said low 
resolution image to a computer processor, said processor 
programmed to carry out the steps of subdividing said low- 
resolution image into groups of pixels; replicating each group of 
pixels as an initial expansion of a restored (high-resolution) 
image; iteratively revising each group of the restored (high- 
resolution) image so as to minimize the BSA score for said group 
of pixels; and providing as an output a restored image. 

9 A machine as in claim 8 wherein said computer processor 
further includes a preprocessing module which provides an output 
to a bimodal estimation module which outputs an estimation to a 
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resolution expansion module for pixel replication and output to a 
BSA restoration module which produces an output representative of 
an enhanced image. 

10. A machine as in claim 8 wherein said BSA restoration module 
comprises means for accepting an image as an input, computing an 
initial BSA score associated with said input and producing an 
updated image output therefrom, computing an updated BSA score 
associated with the updated image, comparing the prior computed 
BSA score with the updated BSA score and continuing to produce 
updated image outputs and calculate updated BSA scores until an 
updated BSA score does not differ from its prior computed BSA 
score by more than a predetermined amount. 
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Figur^. BSA restoration ^ftWWW» 
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(a) Original 



(b) Iteration 3 



(c) Iteration 6 




(d) Iteration 10 



Figure^ Iterative BSA restoration example. 
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(a) Oripnal image 



(b) Linear interpolated image 




(c) Cubic spline image 




(d) BSA restored image 



Figure^ Results of grayscale document restoration. 
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(a) Cubic spline image 

Uy £m a region in a j'™°^^^o1and change phases in the 
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7b) CuWc spline OCR results 
Pigu^ tjwnple OCR results for cubic spline. 
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(a) BSA restored image 
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representative 



(a) Original binary image 




(b) Convolved grayscale image 



(c) Restored grayscale image 



representative 

(d) Restored binary image 
Fisurelt Results of binaiy docmnwit restoration. 
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(a) Original image 



LEADERSHIP CHJSIS 

% of stocks ❖ 30% from highs 



NASDAQ 



(b) Cubic spline image 
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(c) BSA-restored image 



Fi-iue 5 Part of enlarged video frame restoration result' 
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